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A  model  of  team  collaboration  was  developed  that  emphasizes  cognitive  aspects 
of  the  collaboration  process  and  includes  the  major  processes  that  underlie  this 
type  of  communication:  (1)  individual  knowledge  building,  (2)  knowledge  inter¬ 
operability,  (3)  team  shared  understanding,  and  (4)  developing  team  consensus. 

This  paper  describes  research  conducted  to  validate  this  model  and  determine  how 
these  processes  contribute  to  team  performance  by  analyzing  two  collaborative 
decisionmaking  tasks.  Team  communications  that  transpired  during  two  complex 
problem  solving  situations  were  analyzed  and  coded.  Data  was  analyzed  for  two 
teams  that  conducted  a  Maritime  Interdiction  Operation  (MIO)  and  four  teams 
that  engaged  in  an  air-warfare  scenario.  The  MIO  scenario  involves  a  boarding 
team  that  boards  a  suspect  ship  to  search  for  contraband  cargo  (e.g.  explosives, 
machinery)  and  possible  terrorist  suspects.  The  air-warfare  scenario  involves 
identifying  air  contacts  in  the  combat  information  center  of  an  Aegis  ship. 

INTRODUCTION 

Military  forces  are  beginning  to  operate  as  a  networked  force,  which  allows  them  to  plan,  decide, 
and  act  collaboratively  and  concurrently  to  accomplish  many  tasks  simultaneously.  These 
collaborative  capabilities  are  expected  to  contribute  to  reducing  the  time  required  to  accomplish 
military  objectives.  Rapid  access  to  current,  accurate,  and  relevant  information,  and  the  ability  to 
engage  in  real-time  collaboration  with  other  decisionmakers  who  are  geographically  distributed, 
have  become  indispensable  elements  of  the  command  and  control  (C2)  planning  and  decision¬ 
making  process.  While  information  access  has  always  been  critical  to  success  in  military 
operations,  the  processes  embodied  in  recently  emerging  military  concepts  (e.g.,  rapid  decisive 
operations)  place  an  even  greater  emphasis  on  having  rapid  access  to  relevant  and  accurate 
information.  These  new  military  concepts  derive  their  power  from  the  effective  linking  or 
networking  of  the  warfighting  enterprise  (Alberts,  Gartska,  and  Stein,  2003).  This  new  way  of 
conducting  business  is  characterized  by  the  ability  of  geographically  dispersed  forces  to  create  a 
high  level  of  shared  awareness  that  can  be  exploited  to  achieve  rapid  decisive  operations. 

While  the  U.S.  has  an  unmatched  ability  to  gather  information  on  the  environment,  the 
adversary,  and  ourselves,  we  currently  lack  the  collaborative  planning  capabilities  (both  mature 
systems  and  practiced  operators)  and  C2  systems  to  use  this  information  to  enable  decision 
superiority.  The  ability  to  quickly  create  and  leverage  superior  knowledge  is  a  critical  aspect  of 
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effective  military  operations.  This  rapid  formulation  of  knowledge  and  understanding  of  the 
battlespace  should  enable  decision  superiority,  reduce  operational  risk,  and  increase  the  pace, 
coherence,  and  effectiveness  of  operations. 

The  need  for  rapid  access  to  current,  relevant,  and  accurate  information  is  at  an  all-time  premium 
—  especially  for  military  operations.  Moreover,  the  need  for  expeditious  transfonnation  of  that 
information  into  “actionable”  knowledge  is  increasingly  recognized  by  the  warfighter.  This 
exchange  and  transformation  of  information  to  support  the  military  decisionmaker  is  facilitated 
by  a  shared  infonnation  environment,  and  the  tools  that  enable  collaboration.  New  Information 
Technology  (IT)  Tools,  used  as  part  of  a  networked,  web-based  collaborative  system  for 
command  and  control  are  providing  enhanced  capabilities  for  improved  decisionmaking.  These 
tools  support  planning  and  operational  processes  by  providing  an  alternative  means  to 
communicate,  collaborate,  and  share  infonnation  among  operators  and  decisionmakers  than  were 
provided  in  past  operational  environments. 

Concepts  such  as  information  and  knowledge  superiority,  knowledge  management,  and  effects- 
based  operations,  are  important  enablers  of  effective  military  operations.  The  need  to  accomplish 
missions  efficiently  and  effectively,  with  coordinated  action,  points  to  the  demand  for  powerful, 
reliable,  and  capable  IT  tools  to  support  the  military  decisionmaker.  These  tools  are  expected  to 
be  critical  elements  of  success  for  the  decisionmaker  who  will  be  operating  in  a  constrained 
battlespace,  working  toward  the  goals  of  achieving  shared  awareness,  infonnation/  decision 
superiority,  unity  of  effort,  and  the  ability  to  respond  rapidly  and  autonomously. 

Collaborative  Tasks 

Inherent  in  many  problem  solving  tasks  is  the  need  for  extensive  collaboration  and  coordination 
across  functional  areas  and  components  within  the  organization  to  accomplish  the  mission. 
Collaboration  tool  suites  are  being  introduced  to  facilitate  these  information-intensive 
interactions  to  support  operational  planning  and  decisionmaking  processes  by  providing  an 
alternative  means  to  communicate,  collaborate,  and  share  information  among  warfighters  that 
extends  what  is  available  in  today’s  current  operational  environments.  Enabled  by  high-speed 
bandwidth  connectivity  and  electronic  collaboration  tools,  it  is  anticipated  that  a  collaborative 
information  environment  will  facilitate  the  exchange  of  information  among  members  of  the  Joint 
Force  and  those  organizations  supporting  or  being  supported  by  the  Joint  Force.  The  long-tenn 
goal  for  operating  in  a  collaborative  environment  is  to  reduce  planning  timelines  while 
increasing  organizational  effectiveness. 

The  majority  of  military  and  business  tasks  today  are  performed  by  teams  who  collaborate  to 
share  information  and  task  perspectives  in  order  to  reach  a  decision.  Benefits  afforded  by 
collaboration  tools  that  are  especially  germane  to  the  military  include  allowing  smaller  deployed 
warfighter  footprints,  and  capitalizing  on  the  synergy  of  the  total  command  and  control 
infrastructure.  Collaborative  tools  offer  the  added  capabilities  of  providing  the  ability  to  share 
information  and  resources,  and  coordinate  among  individuals  across  geographic  and  temporal 
boundaries.  Collaboration  is  also  essential  to  developing  shared  situational  awareness  among 
heterogeneous,  distributed  team  members. 
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Collaboration  offers  great  potential  to  better  enable  decisionmakers  and  operators  to  plan, 
monitor,  execute,  and  assess  activities  across  the  spectrum  of  activities.  Capabilities  afforded  by 
collaboration  tools  include  the  ability  to  share  applications,  have  a  virtual  workspace,  use 
voice/audio/video,  etc.  Many,  if  not  all,  of  the  benefits  of  participating  in  a  face-to-face  meeting 
can  be  gained  using  collaborative  tools:  Infonnation  flows  quickly,  outstanding  issues  are  raised, 
and  a  certain  amount  of  brainstonning  can  occur  to  arrive  at  a  decision.  Additionally,  all  relevant 
users,  or  providers  of  information,  reach  a  fuller  understanding  of  the  issues  because  they  have 
seen  other  viewpoints  and  received  a  freer  flow  of  information  (Truver,  2001). 

From  a  military  perspective,  advantages  of  using  a  collaborative  environment  include:  fewer 
personnel  have  to  be  located  in  the  area  of  conflict;  there  are  enhanced  opportunities  to  share 
information  among  planners  and  decisionmakers;  experts  in  remote  locations  can  participate  in  all 
phases  of  the  planning,  decisionmaking,  and  assessment  process;  it  will  increase  access  to  many 
additional  sources  of  information  that  previously  were  not  possible;  it  should  reduce  the  time 
required  for  the  planning,  decisionmaking,  and  assessment  process. 

Team  collaboration  and  decision-making  in  complex,  data-rich  situations  is  being  investigated  to 
better  understand  the  cognitive  processes  employed  when  teams  collaborate  to  solve  problems. 
This  paper  reports  on  research  conducted  under  sponsorship  of  the  Office  of  Naval  Research 
(ONR)  Collaboration  and  Knowledge  Interoperability  (CKI)  program.  The  research  reported  in 
this  paper  applies  definitions  of  the  cognitive  processes  included  in  the  model  to  two  different 
decisionmaking  domains.  Both  involve  team  collaboration  to  solve  complex  problems.  The 
Maritime  Interdiction  Operation  (MIO)  task  involves  a  Coast  Guard  Operation  to  search  a 
suspect  ship  for  contraband  material  and  suspect  persons.  The  air  warfare  task  involves  a  US 
Navy  Aegis  cruiser  combat  information  center  team  identifying  air  tracks  in  a  Persian  Gulf 
scenario. 

Model  of  Team  Collaboration 

A  cognitive  model  of  team  collaboration  emphasizing  the  human  decisionmaking  processes  used 
during  team  collaboration  was  developed  by  Warner,  Letsky,  &  Cohen  (2004).  This  model 
applies  to  collaborative  problem  solving  and  includes  the  major  processes  that  underlie 
collaborative  team  problem  solving,  as  depicted  in  Figure  1.  These  processes  include  (1) 
individual  knowledge  building,  (2)  developing  knowledge  inter-operability,  (3)  team  shared 
understanding,  and  (4)  developing  team  consensus.  In  this  paper  we  describe  research  conducted 
to  validate  the  model  and  determine  how  these  processes  contribute  to  team  performance  by 
analyzing  two  complex  decisionmaking  tasks. 

Many  definitions  of  collaboration  are  found  in  the  research  literature,  depending  on  the 
researchers  discipline  and  perspective.  At  the  most  fundamental  level,  collaboration  refers  to  the 
joint  effort  of  two  or  more  agents  to  achieve  a  common  goal  (Nosek,  2003),  where  collaboration 
members  construct  judgments  and  then  act  on  these  judgments.  A  different  definition  of 
collaboration  is  “the  mental  aspects  of  joint  problem  solving  for  the  purpose  of  achieving  a 
shared  understanding,  making  a  decision,  or  creating  a  product.”  Yet  another  definition  states 
that  “collaboration  occurs  when  a  group  of  autonomous  stakeholders  of  a  problem  domain 
engage  in  an  interactive  process,  using  shared  rules,  norms,  and  structures,  to  act  or  decide  on 
issues  related  to  that  domain”  (Wood  and  Gray,  1991). 


4 


The  types  of  problem-solving  situations  this  model  describes  are  ill-structured  decisionmaking 
tasks,  characterized  by  time  pressure,  dynamic  information,  with  high  information  uncertainty, 
high  cognitive  workload  (i.e.,  a  large  amount  of  knowledge  is  brought  to  bear  to  solve  complex 
problems),  and  human-system  interface  complexity.  The  model  focuses  on  three  tasks;  (1)  team 
data  processing,  (2)  developing  a  shared  understanding  among  team  members,  and  (3)  team 
decisionmaking  and  course  of  action  selection.  The  model  consists  of  general  inputs  (e.g.,  task 
description),  collaborative  stages  that  the  team  goes  through  during  the  problem  solving  task,  the 
cognitive  processes  used  by  the  team  and  final  team  outputs,  such  as  the  selected  course  of 
action. 


Problem  Area 
Characteristics 


Collaborative  Situation 


Collaboration  Stages  &  Cognitive  Processes 


•  time  pressure 

•  information/knowledge 
uncertainty 

•  dynamic  information 

•  large  amount  of  knowledge 
(cognitive  overload) 

•  human-agent  interface 


!  Meta-Cognitive: 

j  •  individual  conversion  of 
i  data  to  knowledge 


•  asynchronous 

•  distributed 

•  culturally  diverse 

'  heterogeneous  knowledge 

•  unique  roles 

•  command  structure 
(hierarchical  vs.  flat) 

•  rotating  team  members 

Operational  Tasks 


•  team  decision  making,  COA 
selection 

•  develop  shared  understandir 

•  intelligence  analysis 
(team  data  processing) 


Macro-Cognitivp: 

•  individual  mental  model 
construction 

•  knowledge  interoperability 
development 

•  individual  task,  team  and  domain 
knowledge  development 

•  individual  knowledge  object 
development 

•  individual  visualization  and 
representation  of  meaning 


•  team  integration  of  individual  I  •  team  agreement  on  a  common  .  soiutjon  adjustment  to 
knowledge  for  common  understanding  solution  j  .... 

i  tit  goals  and  exit  criteria 


•  knowledge  interoperability 
development 

•  iterative  information  collection 

and  analysis 

•  team  shared  understanding 
development 

•  develop,  rationalize,  &  visualize 
solution  alternatives 

•  convergence  of  individual  mental 
models  to  team  mental  model 

•  individual  task,  team  and  domain 
knowledge  development 


•  team  negotiation  of  solution  | 
alternatives 

•  team  pattern  recognition 

•  team  shared  understanding  I 
development 

•  convergence  of  individual  I 

mental  models 

•  critical  thinking 

•  sharing  hidden  knowledge  j 

•  individual  task  knowledge  J 
development 

•  team  task  knowledge 
development 


•  compare  problem  solution! 
against  goals 

•  team  shared  understanding 
development 

•  convergence  of  individual 
mental  models  of  solution 

•  analyze,  revise  output 
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Mechanisms  for  achieving  Meta  and  Macro  Cognitive  Processes  (applies  to  all  stages) 

•  Verbal  communications:  representing  and  discussing  individual  information,  discussing  team  generated  information, 
questioning,  agreeing  /  disagreeing,  negotiating  perspectives,  discussing  possible  solutions,  providing  rationale. 

•  Non-Verbal  communications:  facial  expressions,  voice  clues  (vocal  paralanguage),  hand  gestures,  body  movements(kinesics) 


Figure  1.  Model  of  Team  Collaboration.  (From  Warner,  Letsky,  &  Cowan,  2004). 


Team  Types.  Team  types  described  by  the  model  include  teams  who  operate  asynchronously, 
whose  members  are  distributed,  and  culturally  diverse,  where  members  possess  heterogeneous 
knowledge,  due  to  the  unique  roles  played  by  each  team  member,  and  operate  in  a  hierarchical 
organizational  command  structure,  and  in  some  situations  involve  rotating  team  members 
(Warner,  et  al.,  2004).  Members  of  both  the  boarding  party  and  air  warfare  teams  each  have 
distinct  roles  and  bring  their  respective  expertise  (e.g.,  radiological  detection)  to  bear,  and 
combine  their  heterogeneous  knowledge. 

Four  unique  but  interdependent  stages  of  team  collaboration  are  included  in  the  model.  As 
depicted  in  Figure  1,  the  stages  include  knowledge  construction,  collaborative  team  problem 
solving,  team  consensus,  and  outcome  and  evaluation  and  revision.  Cognitive  processes  within 
each  stage  are  represented  at  two  levels:  meta-cognitive  processes,  which  guide  the  overall 
problem-solving  process,  and  macro-cognitive  processes,  which  support  team  members’ 
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activities  within  the  respective  collaboration  stage.  The  model’s  macro-level  definition  of  the 
cognitive  processes  permits  empirical  assessment  of  these  cognitive  processes  with  currently 
available  measurement  techniques  (e.g.,  verbal  protocol  analysis,  communication  analysis). 
Analysis  of  data  captured  from  teams  performing  their  tasks  in  a  collaborative  environment  can 
provide  valuable  insight  into  what  constitutes  effective  collaboration  performance. 

Knowledge  construction  begins  with  team  members  building  individual  task  knowledge  and  the 
construction  of  team  knowledge.  Knowledge  represents  a  pattern  that  connects  and  generally 
provides  a  high  level  of  predictability  regarding  what  is  described  or  what  will  happen  next.  The 
focus  of  all  the  macro-level  cognitive  processes  in  the  knowledge  construction  stage  is  to  support 
individual  and  team  knowledge  development.  This  knowledge  will  be  used  during  collaborative 
team  problem  solving  sessions  to  develop  solution  alternatives  to  the  problem. 

During  collaborative  team  problem  solving  sessions,  team  members  communicate  data,  infor¬ 
mation  and  knowledge  to  develop  solution  options  to  the  problem  (Bellinger,  Castro,  &  Mills, 
2004).  The  majority  of  collaboration  occurs  during  this  stage  (Warner,  et  ah,  2004).  The  focus 
of  the  macro-cognitive  processes  during  this  stage  is  to  support  development  of  solution  options 
for  the  collaborative  problem. 

During  team  consensus  the  team  negotiates  solution  options  and  reaches  final  agreement  by  all 
team  members  on  a  specific  option.  The  macro-cognitive  processes  support  the  team  in  reaching 
total  agreement  on  the  final  solution  to  the  problem.  During  the  outcome,  evaluation  and 
revision  stage  the  team  evaluates  the  selected  solution  option  against  the  problem-solving  goal 
and  revises  the  solution  option  if  that  option  does  not  meet  the  goal. 

METHOD 

Verbatim  transcripts  were  analyzed  from  two  series  of  experiments  where  teams  collaborated  to 
solve  a  complex  problem.  These  decisionmaking  domains  included  Maritime  Interdiction 
Operations  (MIO)  and  air  warfare  decisionmaking  scenarios.  In  both  of  these  problem-solving 
tasks,  assessment  is  particularly  difficult  because  the  available  infonnation  is  often  incomplete  or 
ambiguous.  Transcripts  included  communications  that  occurred  between  all  team  members  as 
well  as  with  decisionmakers  at  the  distributed  sites.  Our  approach  was  to  analyze  and  code  team 
communications  data  using  the  cognitive  process  definitions  developed  by  Warner,  et  ah,  (2004). 
The  focus  of  the  collaboration  model  is  on  knowledge  building  among  the  team  members  and 
developing  team  consensus  for  selection  of  a  course  of  action.  This  research  builds  on  previous 
work  to  validate  this  model  (Warner,  et  al,  2004).  The  current  effort  uses  a  similar  methodology 
applied  to  two  different  decisionmaking  scenarios. 

Experiment  I:  Maritime  Interdiction  Operations 

An  experiment  was  conducted  to  test  the  technical  and  operational  challenges  of  developing  a 
global  Maritime  Domain  Security  testbed.  One  goal  was  to  test  the  applicability  of  using  a 
wireless  network  for  data  sharing  during  a  Maritime  Interdiction  Operations  (MIO)  scenario  to 
facilitate  expert  reach  back  for  radiation  source  analysis  and  biometric  data  analysis.  This 
technology  aims  to  provide  networking  solutions  for  maritime  interdiction  operations  where 
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subject  matter  experts  at  geographically  distributed  command  centers  collaborate  with  a  boarding 
party  in  near  real  time  to  facilitate  situational  understanding  and  course  of  action  selection. 

The  objective  of  this  experiment  was  to  evaluate  the  use  of  networks,  advanced  sensors,  and 
collaborative  technology  for  rapid  MIO.  Specifically,  the  ability  for  a  boarding  party  to  rapidly 
set-up  ship-to-ship  communications  that  permit  them  to  search  for  radiation  and  explosive  sources 
while  maintaining  contact  with  the  mother  ship,  command  and  control  organizations,  and 
collaborating  with  remotely  located  sensor  experts. 

The  boarding  team  boards  the  suspect  vessel  and  establishes  a  collaborative  network  and  then 
begins  their  respective  inspections  and  data  collection  processes.  The  boarding  officer  boards 
the  vessel  with  his  laptop  so  he  can  collaborate  with  all  other  members  of  the  team.  This 
includes  those  who  are  located  on  the  ship,  but  are  physically  spread  out  around  different  areas 
of  the  ship  (while  searching  for  contraband  material  and  obtaining  fingerprints  of  crew 
members),  as  well  as  the  virtual  members  of  the  boarding  team  -  the  experts  who  are  located  at 
the  different  reach  back  centers.  Since  there  are  numerous  commercial  uses  for  certain 
radioactive  sources,  positive  identification  of  the  source  in  a  short  time  is  imperative.  There  is 
also  pressure  to  conduct  the  MIO  quickly  so  as  to  not  detain  the  ship  any  longer  than  necessary. 

MIO  Team  Members.  Members  of  the  boarding  team  include  the  following  team  members:  1) 
the  Boarding  Officer,  a  Coast  Guard  officer;  2)  a  representative  from  Lawrence  Livennore 
National  Labs  (LLNL)  with  portable  radiation  detection  devices  and  “reach-back”  capability  to 
LLNL;  3)  a  representative  from  the  Defense  Threat  Reduction  Agency  (DTRA),  who  uses 
biometrics  measurements  of  fingerprints  and  video  imagery  to  be  checked  against  databases  at 
the  remote  facility;  and  4)  a  representative  from  Special  Operations  Command  (SOCOM),  who 
provides  guidance  on  handling  hazardous  material. 

Maritime  Interdiction  Operations  Scenario.  Based  on  intelligence,  the  US  Coast  Guard  has 
ordered  one  of  its  cutters  to  stop,  board,  and  search  a  commercial  vessel  of  foreign  origin 
suspected  of  transporting  uranium  enriching  equipment.  The  boarding  party  brings  radiation 
detection  and  biometric  gear,  drawings  of  dangerous  equipment  and  people,  and  video  recording 
capability.  Data  is  collected  on  suspicious  material,  equipment,  and  people  and  sent  to  specific 
experts  at  distributed  reach  back  centers.  A  network  extension  capability  was  utilized  from  the 
cutter  to  the  boarding  team;  the  network  was  able  to  reach  back  to  LLNL  and  DTRA  to  assist  in 
identification  of  suspect  cargo.  Support  from  the  National  Biometric  Fusion  Center  was  used  to 
quickly  and  accurately  discriminate  between  actual  vessel  crewmembers  and  non-crew  suspect 
persons. 

The  Groove  collaborative  workspace  brought  expert  services  into  the  boarding  party  team’s  tool 
set  and  facilitated  voice  and  text  communications  between  all  members  of  the  virtual  boarding 
party  and  physical  boarding  party.  Remote  sites  were  able  to  receive  and  open  posted  files  in  less 
then  two  minutes  to  begin  their  analysis.  For  example,  expert  services  provided  at  LLNL  quickly 
determined  the  need  for  additional  data  capture  of  longer  length  and  different  angles  of  approach. 
Requests  were  transmitted  by  text  message  and  taken  for  action,  and  radiation  source  spectrum 
captures  were  made  of  suspect  containers  that  were  detected  to  have  a  radiation  signature 
presence.  Analysis  of  this  data  led  the  boarding  officer  to  recommend  that  the  vessel  be 
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quarantined  for  further  inspection.  The  biometric  team  took  digital  prints  of  the  crew  to  be 
compared  to  known  criminal  prints  and  latent  prints  from  terrorist  and  crime  scenes. 

Cognitive  Complexity  of  Scenarios.  Scenarios  used  for  this  research  focus  on  detecting, 
identifying,  and  interdicting  nuclear  materials  in  open  waters.  The  critical  task  involves  the 
cognitively  complex  issue  of  discrimination,  that  is,  how  to  detennine  the  presence  of  contraband 
radiological  material  against  a  background  containing  multiple  benign  radiation  sources. 
“Smoke  detectors,  radiant  signs,  and  a  container  load  of  bananas  all  share  the  ability  to  be  moved 
in  commercial  vehicles  or  vessels... and  all  three  can  cause  radiation  detectors  to  alarm.” 
(Schwoegler,  206,  p.4).  For  example,  “smoke  detectors  contain  small  amounts  of  americium, 
radiant  signs  glow  because  they  contain  tritium,  a  radioactive  hydrogen  isotope,  and  bananas, 
contain  a  small  fraction  of  potassium-40  which  emits  ionizing  radiation.”  (ibid,  p.4). 

Technical  expertise,  provided  by  remotely-located  experts,  is  required  to  interpret  the  scientific 
signals  emitted  from  complex  detectors  to  enable  on-site  personnel  to  make  the  line 
discriminations  required.  Performing  these  complex  discriminations  is  made  possible  by  the 
collaborative  capability  provided  by  the  collaborative  workspace  in  terms  of  bringing  remote 
expertise  to  the  vessel  undergoing  the  search  and  the  ability  to  rapidly  send  and  receive 
communications  between  a  diverse  team  of  experts  who  all  bring  their  respective  expertise  to 
bear  with  a  potentially  high-threat  situation.  The  search,  identification,  and  final  decisions  need 
to  be  conducted  rapidly  as  the  economic  and  political  ramifications  of  detaining  a  commercial 
cargo  vessel  are  great.  On  a  commercial  vessel  that  is  under  way,  false  positives  can  prove 
economically  costly  and  politically  embarrassing. 

Detecting  a  moving  vessel  emitting  signs  of  ionizing  radiation  involves  initial  detection  by  a 
local  police  maritime  unit.  This  initial  detection  then  triggers  the  need  for  Coast  Guard  officers 
to  board  the  vessel  and  take  in-depth  readings  with  portable  radiation-detection  instruments. 
These  readings  are  immediately  relayed  to  scientific  experts,  at  geographically  distributed 
locations,  and  the  analyzed  results  are  electronically  sent  back  to  the  boarding  vessel  for  use  by 
first  responders  on  the  scene. 

Experiment  II:  Air  Warfare  Decisionmaking 

Air  warfare  decisionmaking  is  conducted  in  the  combat  information  center  of  a  Navy  ship.  The 
team  has  responsibility  for  identification  of  a  large  number  of  air  tracks  under  high  time  pressure. 
These  air  tracks  can  fit  multiple  hypotheses  regarding  the  level  of  threat  they  pose  to  the 
battlegroup  due  to  the  high  level  of  ambiguity  associated  with  the  data.  The  nature  of  the  data, 
the  complex  judgments  required,  and  a  sociotechnical  environment  that  is  characterized  by  high 
workload,  and  high  stakes,  all  combine  to  create  an  extremely  challenging  problem  for  the  air 
warfare  team. 

Incoming  information  arrives  via  various  sensor  systems  (radar,  electronic  support  measures 
system,  identification  friend  or  foe,  etc.),  and  various  reports,  e.g.,  intelligence  reports,  other 
platforms  in  the  area  pass  messages  regarding  situation  reports  on  various  tracks,  and  so  on.  All 
these  reports  are  passed  by  the  team  member  who  operates  that  sensor,  or  who  receives  the 
message,  to  the  rest  of  the  team  over  any  of  several  communications  systems.  These  reports  are 
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generally  heard  by  all  other  team  members  as  they  are  all  on  the  same  communications  net, 
although  the  reports  are  typically  addressed  to  a  specific  team  member/s,  and  sometimes  they  are 
addressed  to  “all.”  The  two  key  decisionmakers  are  the  commanding  officer  and  the  tactical 
action  officer. 

Reports  on  specific  tracks  are  interleaved  with  reports  on  other  tracks.  Communications  between 
team  members  are  passed  as  soon  as  new  information  is  received  and  updated  reports  are  passed 
as  soon  as  new  information  is  obtained  for  any  track.  So,  for  example,  in  a  series  of  speech 
turns,  five  separate  contacts  may  be  discussed  at  various  levels  -  initial  reports,  updated  reports, 
sharing  information  on  the  response,  or  lack  of  response,  by  the  contact  to  some  action  taken  by 
the  ship,  etc.  Five  consecutive  reports  could  pertain  to  five  separate  tracks  or  all  reports  could 
pertain  to  one  track. 

Air  Warfare  Team  Members.  Six  collocated  team  members  consisted  of  the  commanding 
officer  (CO),  tactical  action  officer  (TAO),  air  warfare  coordinator  (AAWC),  electronic  warfare 
supervisor  (EWS),  identification  supervisor  (IDS)  and  tactical  information  coordinator  (TIC). 
These  combat  infonnation  center  team  members  also  communicated  with  several  non-collocated 
information  sources,  e.g.,  the  battle  group  commander,  the  Saudi  air  tower,  assets  passing 
intelligence  reports,  other  ships  and  friendly  aircraft  in  the  vicinity  of  the  battlegroup,  to  gather 
additional  information  from  them  and  keep  them  apprised  of  the  unfolding  scenario  as  they 
collaborated  to  identify  air  tracks. 

Air  Warfare  Decisionmaking  Scenario.  The  global  air  warfare  task  involves  identification  and 
responding  to  numerous  contacts.  When  an  aircraft  (or  a  surface  contact)  is  detected,  CIC 
personnel  work  as  a  team  to  determine  the  identity  and  to  try  to  determine  whether  or  not  the 
aircraft  poses  a  threat.  The  high  degree  of  inherent  ambiguity  associated  with  contact  information 
can  often  make  threat  assessment  a  very  difficult  task.  This  is  because  many  pieces  of  data  fit 
multiple  hypotheses  regarding  threat  assessment.  The  global  response  choices  (that  is,  engage, 
monitor,  do  nothing)  are  largely  determined  by  the  ship’s  orders  and  the  current  geopolitical 
situation.  Specific  actions  (such  as,  change  course,  issue  verbal  warnings,  illuminate  with  radar, 
challenge  with  other  sensors,  etc.)  depend  on  the  local  conditions  and  the  relative  positions  of  the 
inbound  contact  of  interest  an  own-ship.  Determining  which  of  these  actions  is  likely  to  be 
effective  depends  on  maintaining  an  accurate  threat  assessment  which  requires  continually 
updating  based  on  iterative  situation  assessments. 

Critical  air  contacts  are  identified  based  on  ambiguous  information  under  time  pressure  to 
determine  if  the  track  posed  a  threat  to  the  ship.  One  of  the  most  challenging  aspects  of  the 
combat  infonnation  center  teams’  job  is  the  high  mental  workload  that  is  entailed  when  a 
constant  stream  of  information  must  be  continuously  evaluated,  particularly  when  the 
information  often  pertains  to  several  different  air  contacts  (or  “tracks”).  Relevant  data / 
information  items  must  be  associated  with  the  right  track  number,  then  analyzed,  synthesized  and 
aggregated.  This  task  places  an  extremely  high  load  on  working  memory.  The  air  warfare  team 
must  assess,  compare,  and  resolve  conflicting  information,  while  making  difficult  judgments  and 
remembering  the  status  of  several  evolving  situations.  These  tasks  are  interleaved  with  other 
tasks,  such  a  making  reports  to  higher  authority  and  requesting  assets. 
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In  general,  the  overall  task  of  responding  to  air  warfare  scenarios  consists  of  situation  assessment 
(“what’s  going  on”)  and  action  selection  (“what  to  do  about  it”).  Recent  theories  of  decision¬ 
making  emphasize  the  importance  of  situation  assessment  for  good  decisionmaking  in 
naturalistic,  event-driven  situations  (Hutchins,  1995).  Moreover,  they  stress  that  decisions 
regarding  actions  to  be  taken  are  a  by-product  of  developing  the  situation  awareness  that 
precedes  action  selection. 

Coding  Process 

Cognitive  process  coding  definitions  developed  by  Warner,  et  al.  (2004)  were  used  to  code  all 
speech  turns.  The  coders  attempted  to  develop  criteria  for  applying  the  coding  schema  as  a 
number  of  coding  categories  appear  to  have  similar  meanings.  This  codification  of  the  coding 
process  is  part  of  the  overall  validation  of  the  model,  in  that  one  goal  is  to  have  high  inter-rater 
reliability  between  coders.  It  was  important  to  pay  attention  to  which  track  a  team  member  was 
talking  about  when  coding  the  speech  turns.  (This  could  sometimes  be  challenging  because  the 
last  time  a  track  was  discussed  may  have  several  pages  previously  in  the  transcript  and 
sometimes  the  person  speaking  did  not  always  state  the  track  number  that  was  being  discussed. 
In  these  situations  the  rater,  by  careful  reading  of  the  communications,  was  able  to  infer  which 
track  the  speech  turn  referred  to.) 

The  first  time  they  discussed  a  track  the  speech  turn  was  coded  as  a  2  ( individual  mental  model 
(IMM)  construction  -  where  an  individual  team  member,  using  available  infonnation,  develops 
his/her  mental  picture  of  the  problem  situation).  After  three  speech  turns  that  discussed  the  same 
track  (typically  involving  at  least  four,  of  the  six  or  more  team  members)  it  was  coded  as  a  4 
(team  knowledge  development  (TKM)  -  where  all  team  members  participate  in  clarifying 
information  to  build  team  knowledge.  Once  five-six  team  members  had  discussed  a  track,  and  at 
least  4  of  the  6  team  members  had  been  involved  in  discussing  this  particular  track,  it  was  coded 
as  a  10  -  team  shared  understanding  development  -  which  includes  discussion  among  all  team 
members  on  a  particular  topic  or  data  item. 

Some  exceptions  to  the  above  mentioned  coding  criteria  include:  When  a  team  member 
addresses  “All  stations,  [track  #  7010  is  a  comm-air.]”  this  means  he  is  telling  all  team  members 
this  evaluation  of  the  track.  Because  it  was  addressed  to  all  team  members  and  it  reported  a 
higher  level/  more  final  assessment  of  the  track,  i.e.,  it  is  a  commercial  airliner,  this  was  coded  as 
a  10.  As  more  team  members  get  involved  in  discussing  a  contact  (i.e.,  more  reports  and/or 
updates  have  been  shared  among  team  members,  the  cognitive  process  coding  category  reflects  a 
higher  level  of  team  understanding  of  the  situation  regarding  that  particular  track. 

RESULTS 

Table  1  presents  the  cognitive  process  definitions  developed  for  the  model  of  team  collaboration. 
We  added  new  examples  for  each  coding  category,  based  on  our  analysis  of  the  air  warfare 
scenarios,  to  provide  illustrative  examples  of  the  types  of  communications  that  fall  under  each  of 
the  coding  categories.  (The  original  set  of  cognitive  process  definitions  included  examples  from 
a  non-combatant  evacuation  scenario.)  These  examples  of  the  coding  categories  are  contained  in 
Table  1. 
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New  Coding  Categories 

During  the  analysis  phase  new  coding  categories  emerged  when  coding  the  air  warfare  scenarios. 
These  new  categories  include  21-23,  miscellaneous,  issue  an  order  regarding  a  course  of  action, 
and  request  a  person  take  some  action.  Examples  of  these  categories  are  included  in  Table  1 
which  includes  the  original  coding  categories.  Miscellaneous  category  includes  getting  the 
attention  of  a  team  member  (e.g.,  “TAO  this  is  EW”)  prior  to  speaking  (so  as  not  to  waste  time 
speaking  if  the  team  member  is  busy  and  not  ready  to  listen  to  the  message),  acknowledging  a 
request  to  speak,  acknowledging  a  message,  e.g.,  “Copy  all,”  and  issuing  a  verbal  warning  to  the 
potential  threat  track  over  the  radio  system.  Because  standard  operational  procedure  for 
communicating  requires  that  significant  communications  be  acknowledged,  this  wound  up  being 
the  largest  category  of  communications. 

Issuing  an  order  regarding  a  course  of  action  appears  to  be  a  significant  coding  category.  This 
refers  to  situations  where  a  person  with  higher  rank  (e.g.,  commanding  officer  speaking  to  the 
tactical  action  officer  (TAO),  or  the  TAO  speaking  to  one  of  the  enlisted  system  operators,  tells 
them  to  take  some  specific  action  against  a  potential  threat  track.  These  actions  include  issuing 
verbal  warnings,  illuminating  or  locking-on  with  radar,  developing  a  firing  solution,  covering 
with  missiles,  etc.  This  category  also  includes  responding/  reporting  they  have  taken  the  action, 
or  acknowledging  the  order. 

Request  a  team  member  take  some  action  refers  to  telling  a  team  member  to  do  something  but  it 
is  not  a  direct  action  against  a  threat  track.  For  example,  “Can  you  try  and  change  7006  and 
7005  to  assumed  hostile.  I  keep  trying  and  can’t  get  it  to  do  it.” 

Other  potential  new  categories  include  Prodding  a  team  member  to  jog  their  awareness,  e.g.,  to 
make  sure  they  are  following  the  discussion,  or  they  push  or  suggest  to  one  or  more  team 
members  to  go  out  and  generate  knowledge,  e.g.,  “You  should  go  back  and  see  if  there  is  ...”. 
This  person  directing  or  suggesting  might  act  in  a  role  as  teacher  gently  pushing  the  collaborative 
effort  in  a  certain  way.  Another  potential  new  category  is  for  “ Contrarians'’ ’  when  a  person  says 
“Let’s  re-evaluate/  reconsider,  or  when  the  person  disagrees  with  the  current  thinking  of  the 
team.  This  would  be  an  “outlier”  who  makes  the  team  consider  another  viewpoint,  or  “pulls 
back  the  reins.”  As  additional  scenarios  are  analyzed  and  coded  new  categories  may  emerge. 

Table  1.  Cognitive  Process  Definitions 

(From  Warner,  Letsky,  and  Cowan,  2004) 


Cognitive  Process  Definitions 

1. 

Metacognition  dti:  individual  conversion  of  data  to  information  =  individual  team  member 
converting  data  to  information. 

•  “We  have  Don-2  bearing  086  and  LN-66  bearing  O97.”(converted  detected  radar 

parameters  -data  to  information  -  names  of  radars  on  specific  bearings) 

•  “I  am  showing  8044  at  400  knots  and  about  27  thousand  feet,  possible  comm-air  type 

profile.” 

•  “I  have  a  second  contact  at  1000  feet.” 

2. 

Macrocognition  imm:  individual  mental  model  construction  =  individual  team  member,  using 
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available  information,  develops  his/her  mental  picture  of  problem  situation. 

•  “8030  definitely  originated  from  Iranian  airspace?  The  possible  helo?” 

•  “That’s  affirmative,  sir.” 

•  “APQ-120  bears  072  off  possible  Foxtrot  4  Delta  or  Echo.” 

•  “We  have  8053,  that  air  unknown  coming  in  up  there.” 

•  “2017  is  squawking  a  comm-air  mode  3.  In  company  with  2025,  but  that  track  is  much 

lower  than  the  comm-air.  One  at  37000,  one  at  8000  just  came  in  low.” 

3. 

Macrocoenition  itk:  individual  task  knowledge  development  =  individual  team  member  asking 
for  clarification  to  data  or  information;  response  to  clarification. 

•  “Do  we  have  the  track  number  for  his  CAP?  I  would  prefer  to  have  the  track  number  for 

his  CAP.” 

•  “Are  you  covering  with  birds?” 

•  “That’s  affirmative,  sir.” 

•  “Did  you  illuminate  him?”  (clarifying  action  has  been  taken) 

•  “Did  you  establish  communications  with  him?” 

•  “Since  he  is  turning  to  the  east  do  you  still  want  us  to  continue  with  level  one?” 

(clarification  of  actions  to  be  taken) 

•  “That’s  a  negative.”  (response) 

•  “The  Desert  Eagle  don’t  have  that  information  for  you  right  now.  I  asked  them  to  get  that 

for  us.  Whether  the  F-ls  were  clean  or  dirty.” 

4. 

Macrocognition  tk:  team  knowledge  development  =  All  team  members  participate  in  clarifying 
(i.e.  answering  a  question)  information  to  build  team  knowledge. 

•  “Rainbow  is  sending  Desert  Eagle  101  and  Desert  Eagle  102  over  to  investigate  track 

8037  (TN  7034).” 

•  “Fie  looks  like  he  is  on  a  [air]  corridor,  Kuwait  City  to  Bushehr.” 

•  “Received  ESM  of  Cyrano  4  bears  121  off  the  F-l.”  (I)  —  information 

•  “No  response  track  8070.”  (I)  —  information 

•  “I  don’t  have  mode  3  or  any  other  type  of  IFF  available  to  me  right  now.” 

•  “They’re  going  too  fast  for  that.” 

•  “Looks  like  he’s  comm-air,  he’s  high  and  looks  like  a  comm-air  profile.” 

5. 

Macrocognition  ko:  knowledge  object  development  =  pictures,  icons  or  standard  text,  developed 
by  an  individual  team  member  or  the  whole  team,  that  represents  a  standard  meaning  to  the 
team. 

—  [No  coded  examples  for  air  warfare] 

6. 

Macrocognition  vrm:  individual  visualization  and  representation  of  meaning 

Visualization  =  individual  team  members  use  methods  (e.g.,  graphs,  pictures)  to 
transfer  meaning  to  other  team  members. 

Representation  =  individual  team  members  use  methods  to  sort  data  and  information 
into  meaningful  chunks. 

—  [No  coded  examples  for  air  warfare] 

7. 

Metacognition  cu:  team  integration  of  individual  knowledge  for  common  understanding  =  all 

team  members  combine  individual  nieces  of  knowledge  to  achieve  a  common  understanding. 

•  “Track  7005  has  turned  west  and  is  now  inbound,  sir.” 

•  “He’s  holding  in  altitude,  he’s  not  far  from  the  air  way,  he  flew  out  of  good  guys  country 

and  we  have  a  comm-air  radar.  Let’s  make  him  assumed  friend.” 
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8. 

Macrocognition  kio:  knowledge  interoperability  development  =  team  members  exchanging 
knowledge  among  each  other. 

•  “Desert  Eagles  report  “tally  ho”  on  section  of  two  Iranian  F-ls,  out.”  — (derived 
knowledge  from  aircraft  providing  a  visual  identification) 

•  “We  have  Primus  40,  bears  135,  Gulfstream  2,  possible  Super  Puma.” 

•  “It  looks  like  the  AWACs  is  feet  dry.  The  CAP,  composition  2,  appear  to  be  headed  feet 
dry  now.” 

•  “Doctrine  won’t  work  for  2017,  make  unknown  assumed  enemy.” 

9. 

Macrocognition  ica:  iterative  information  collection  and  analysis  =  collecting  and  analyzing 
information  to  come  up  with  a  solution  but  no  specific  solution  mentioned. 

•  “No  response  initial  warning,  track  8037.” 

•  “We  need  a  report  from  CAP  as  to  whether  those,  upon  intercept  of  those  suspected 

Pumas,  whether  they  are  armed  or  not.” 

•  “Track  2017  deviated  from  known  flight  path  still  maintaining  altitude  and  still  squawking 

the  same  mode  3.” 

10. 

Macrocognition  tsu:  team  shared  understanding  development  =  discussion  among  all  team 
members  on  a  particular  topic  or  data  item  (i.e.  discussion  does  not  involve  answering  questions) 

•  “Track  8061  bearing  027  Princeton  at  25  miles,  5000  feet,  heading  south,  covering  with 

birds.” 

•  “It  looks  like  he  is  turning  to  the  west.” 

•  “You  need  to  watch  him  closely  here.” 

•  “Track  8061  appeared  to  originate  from  Iran.  When  we  picked  him  up  he  was  already  off 

the  coast  but  he  was  coming  south  from  close  to  the  Iranian  coast.  I  can’t  confirm  that  he 
came  from  Iran  but  he  was  coming  from  that  direction.” 

•  “I  am  showing  a  CPA  of  43  miles  to  the  south  at  their  current  heading.” 

•  “Cyrano  4,  that  emitter  has  ceased.  Last  bearing  for  Cyrano  4  was  122.” 

•  “OK,  what  do  we  think  about  the  Saudi  CAP?  Shot  down,  too  low  to  communicate?” 

•  “It  looks  like  we  still  have  a  good  track  on  them,  27  at  13000.  Probably  just  poor  comms 

with  the  AWACS.” 

•  “Continue  to  track  sections  of  Iranian  F-ls  and  F-4s.  Approached  the  force  with  an  attack 
profile.  Interrogated  level  1  with  no  response.  They  turned  away  from  the  force  at  a 
range  of  abut  30  miles.  Continuing  to  track.” 

11. 

Macrocognition  sa:  develop,  rationalize  and  visualize  solution  alternatives  =  using  data  to 
justify  a  solution 

•  “I  would  like  fire  control  lock  up  on  7010  and  I’d  like  to  make  sure  he  is  designated  as  a 
gun  target.  I’d  like  to  have  two  rounds  of  illumination  prepped  on  mount  52.” 

•  “My  intentions  are  to  issue  a  warning  shot  with  a  flare  if  the  helo  proceeds  to  within  ten 
nautical  miles,  over.” 

•  “Indicate  to  7010  that  if  he  continues  to  close  he  can  expect  defensive  actions.” 

•  “Track  number  7010  continuing  inbound,  request  permission  to  engage  at  three  nautical 

miles,  no  response  to  all  measures,  so  far.” 

12. 

Macrocognition  cmm:  convergence  of  individual  mental  models  to  team  mental  model  = 

convincing  other  team  members  to  accept  specific  data,  information  or  knowledge 
•  “OK,  we  need  to  make  them  assumed  enemy  and  cover  them  ,  AAWC.” 

13. 

Metacognition  cs:  team  agreement  on  a  common  solution  =  all  team  members  agree  on  the  final 
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plan. 

•  “Listen  up.  8044  is  a  probable  comm-air,  8100  is  an  assumed  hostile.” 

•  “8044  looks  like  a  comm-air  profile.” 

•  “ID  2010  unknown  assumed  friend.” 

•  “Request  batteries  release  on  track  7010,  it  is  continuing  inbound,  he  is  at  three  nautical 
miles,  request  permission  to  engage,  over.” 

14. 

Macrocognition  tn:  team  negotiation  of  solution  alternatives  =  team  negotiation  of  solution 
alternatives  ending  in  a  final  solution  option,  (solution  options  are  defined  for  each  of  the 
five  components  of  the  final  plan  —  i.e.  personnel,  transportation,  weapons,  critical  times  and 
detail  plan) 

—  [No  coded  examples  for  air  warfare] 

15. 

Macrocognition  tpr:  team  pattern  recognition  =  the  team  as  a  whole  identifies  a  pattern  of  data, 
information  or  knowledge. 

—  [No  coded  examples  for  air  warfare] 

16. 

Macrocognition  ct:  critical  thinking  =  Team  working  together  toward  a  common  goal,  whereby 
goal  accomplishment  requires  an  active  exchange  of  ideas,  self-regulatory  judgment,  and 
systematic  consideration  of  evidence,  counterevidence,  and  context,  in  an  environment  where 
judgments  are  made  under  uncertainty,  and  there  is  limited  knowledge  and  time  (Hess  &  Freeman, 
2004). 

1.  critical  thinking  is  measured  as  a  composite  of:  (Warner  &  Wroblewski,  2004; 

Hess  &  Freeman,  2004) 

•  MCitk:  individual  task  knowledge  development  =  individual  team 
member  clarifying  data;  asking  for  clarification. 

•  MetCcu:  team  integration  of  individual  knowledge  for  common 
understanding  =  one  or  more  team  members  combine  individual  pieces 
of  knowledge  to  achieve  a  common  understanding. 

•  MCkio:  knowledge  interoperability  =  team  members  exchanging 
knowledge  among  each  other. 

•  MCsa:  develop,  rationalize  and  visualize  solution  alternatives  =  using 

data  to  justify  a  solution 

Note:  one  critical  thinking  frequencv  count  =  oneMCitk  +oneMetCcu  + 

MCkio  +  MCsa 

17. 

Macrocognition  shk:  sharing  hidden  knowledge  =  individual  team  members  sharing  their 
knowledge  through  prompting  by  other  team  member(s). 

•  “We  still  have  no  level  two  warnings  out  to  those  guys.” 

•  “Yes  sir,  we  ID’d  him  as  a  com[mercial]  earlier,  we  will  go  ahead  and  talk  to  him.” 

•  “I’ve  got  track  7011  ID’d  as  com-air.  He  started  out  at  35,000  feet,  now  he  is  descending.” 

18. 

Metacognition  sag:  solution  adjustment  against  goal  and  exit  criteria  =  team  as  a  whole 
compares  complete  solution  option  against  goal  and  exit  criteria. 

•  “Ceased  illumination  8005,  maintaining  lock  on  8005.  Turning  outbound.” 

19. 

Macrocognition  csg:  compare  solution  options  against  goal(s)  =  team  members  discuss  solution 
options  (i.e.  any  of  the  five  solution  components)  against  the  scenario  goal  (i.e.  rescue  3  red 
cross  workers  within  24  hrs). 

•  “Ah  Rainbow’s  holding  track  number  7011,  low  and  slow  and  inbound.  Do  you  desire  me 
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to  cover  with  birds  also?” 

20. 

Macrocognition  aro:  analyze,  revise  solution  options  =  team  members  analyze  final  solution 
options  (i.e.  any  of  the  five  solution  components)  and  revise  if  necessary. 

—  [No  coded  examples  for  air  warfare] 

21. 

Miscellaneous:  misc  =  acknowledging  a  message,  asking  for  repeat  of  message,  verbal  warning 

•  “Copy  all,  out.” 

•  “What  was  your  last?” 

•  Verbal  warning  issued  to  inboard  aircraft 

22. 

Issue  order  regarding  a  course  of  action:  coa  =  a  superior  in  the  chain  of  command  tells  a  team 
member  to  take  a  specific  action  against  a  possible  threat  track. 

•  “Cover  8032  (TN  7013)  with  standard  missile  also  generate  a  SWG  1A  solution 

on  him.” 

•  “Cease  illumination.” 

•  “Let’s  start  level  ones,  8070.” 

23. 

Request  take  action:  rta  =  team  member  requests  another  team  member  take  some  action. 

•  “Let’ s  investigate  with  CAP.” 

•  “Confirm  that  tracks  originating  from  Iranian  air  space  are  designated  unknown 

assumed  hostile.” 

•  “Have  SWC  develop  a  Harpoon  solution  on  him.” 

•  “Go  ahead  and  tag  8037  as  F-ls.” 

•  “Make  8037  and  company  assumed  hostile.” 

•  “Shift  your  focus  Air  to  8070,  inbound  helo.” 

•  “Increase  speed  as  well.” 

For  the  MIO  scenario,  52%  of  the  speech  turns  contained  content  related  to  solving  the  problem; 
48%  involved  administrative  types  of  communications  Evidence  for  twelve  of  the  twenty-two 
cognitive  processes  included  in  the  model  were  found.  For  the  air  warfare  scenario,  99%  of  the 
speech  turns  contained  content  related  to  solving  the  problem;  1%  involved  administrative  types 
of  communications  Evidence  for  fifteen  of  the  twenty-three  cognitive  processes  included  in  the 
model  were  found.  Multiple  occurrences  for  most  of  these  cognitive  processes  were  found  in 
both  scenarios. 

Table  2  presents  the  cognitive  process  coding  tallies  for  the  four  air  warfare  scenarios.  The  large 
number  of  speech  turns  coded  as  itk  reflects  the  high  degree  of  uncertainty  inherent  in  air  warfare 
decisionmaking  tasks.  An  interesting  example  of  sharing  hidden  knowledge  (17)  occurred  when 
the  CO  issued  an  order  to  issue  a  verbal  warning  and  “lock  up”  the  inbound  aircraft.  The  next 
speech  turn  involved  the  TAO  replying  “Yes,  sir,  we  identified  him  as  a  com[mercial  aircraft] 
earlier,  we  will  go  ahead  and  talk  to  him.”  In  this  situation,  the  TAO  was  gently  reminding  the 
CO  of  a  critical  piece  of  information  that  he  had  forgotten.  The  large  number  of  speech  turns 
coded  as  categories  1-4  reflects  the  huge  emphasis  on  individual  knowledge  construction  for  the 
air  warfare  task.  Similarly,  that  examples  of  all  six  categories  in  the  collaborative  team  problem 
solving  phase  -  where  teams  integrate  individual  knowledge  for  common  understanding  -  had 
many  speech  turns  coded  as  these  categories  also  indicates  the  large  role  these  cognitive 
processes  play  for  air  warfare  teams.  In  contrast,  the  small  percentage  of  speech  turns  that  were 
coded  as  cognitive  processes  associated  with  team  consensus  and  outcome  evaluation  and 
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revision  indicated  that  the  course  of  action  selection  phase  of  air  warfare  is  not  conducted  in  a 
collaborative  manner. 


Table  2.  Cognitive  Process  Coding  Tallies  for  Air  Warfare  and  MIO  Scenarios. 


Air  Warfare 

Maritime 

Interdiction 

Macro-Cognitive  Process  Coding 
Categories 

Seen 

D-A 

Seen 

D-B 

CG- 

59 

DDG- 

54 

Nov 

06 

June 

06 

Sept 

06 

Knowledge  Construction 

1. 

Data  to  infonnation  (dti) 

1 

4 

- 

37 

2 

5 

- 

2. 

Individual  mental  model  (imm) 

8 

11 

18 

25 

1 

7 

8 

3. 

Individual  task  knowledge 
development  (itk) 

25 

30 

31 

29 

35 

7 

47 

4. 

Team  knowledge  development  (tk) 

11 

5 

18 

1 

3 

5 

8 

5. 

Knowledge  object  development  (ko) 

- 

- 

- 

- 

- 

2 

8 

6. 

Visualization  and  representation  (vrm) 

- 

- 

- 

- 

- 

- 

- 

Collaborative  Team  Problem  Solving 

7. 

Common  understanding  (cu) 

- 

6 

- 

2 

6 

7 

8. 

Knowledge  interoperability  (kio) 

- 

5 

- 

1 

2 

- 

10 

9. 

Iterative  collection  and  analysis  (ica) 

1 

11 

- 

- 

6 

4 

14 

10. 

Team  shared  understanding  (tsu) 

1 

17 

28 

34 

3 

2 

3 

11. 

Solution  alternatives  (sa) 

- 

3 

- 

- 

6 

- 

- 

12. 

Convergence  of  mental  models  (cmm) 

1 

- 

- 

- 

1 

- 

- 

13. 

Agreement  on  Common  solution  (cs) 

- 

2 

- 

- 

- 

- 

Team  Consensus 

14. 

Team  negotiation  (tn) 

- 

- 

- 

- 

4 

- 

- 

15. 

Team  pattern  recognition  (tpr) 

- 

- 

- 

- 

- 

- 

- 

16. 

Critical  thinking  (ct) 

- 

- 

- 

- 

- 

- 

- 

17. 

Sharing  hidden  knowledge  (shk) 

- 

2 

- 

- 

- 

- 

- 

18. 

Solution  adjustment  against  goal  (sag) 

- 

- 

- 

- 

- 

- 

Outcome  Evaluation  and  Revision 

19. 

Compare  solution  options  against  goals 
(csg) 

1 

20. 

Analyze,  revise  solutions  (aro) 

- 

- 

- 

- 

- 

- 

- 

21. 

Miscellaneous  (misc) 

38 

27 

57 

61 

6 

- 

- 

22. 

Issue  order  regarding  course  of  action 
(coa) 

7 

5 

17 

37 

“ 

2 

23. 

Request  take  action  (rta) 

3 

2 

18 

8 

1 

2 

11 

Totals 

96 

131 

187 

233 

73 

40 

118 

16 


Table  3  presents  an  excerpt  of  the  communications  coding  from  the  MIO  scenario  where  the 
team  is  developing  solution  alternatives  by  using  data  to  justify  a  solution.  First  (1),  individual 
TMs  are  clarifying  data  regarding  the  degree  of  danger  inherent  in  the  material  discovered  (2) 
and  exchanging  knowledge  among  each  other,  i.e.,  the  material  needs  to  be  confiscated  (3), 
based  on  infonnation  provided  by  one  of  the  remote  centers  (the  material  needs  to  be  handled 
carefully).  An  individual  exchanges  knowledge  with  other  TMs  (4)  to  develop  knowledge 
interoperability  regarding  whether  the  Coast  Guard  ship  has  a  suitable  storage  area  for  the 
confiscated  material  (5).  Finally,  TMs  combine  individual  pieces  of  knowledge  to  achieve  a 
common  understanding  (6)  regarding  the  next  action  to  be  taken. 

Table  3.  Excerpt  from  MIO  Scenario  Communications  Coding: 

Developing  Solution  Alternatives. 


MIO  Team  Communications 

Cognitive  Process  Coding 

Speaker 

Code 

1 

DTRA 

Cesium  137  can  be  used  to  make  an  RDD. 
If  there  are  no  explosives,  then  it  is  not 
configured  as  a  weapon  yet.  Recommend 
material  be  confiscated. 

MCsa 

Develop,  rationalize  and  visualize 
solution  alternatives',  using  data  to 
justify  a  solution 

2 

BO 

Roger  will  confiscate. 

MCitk 

Individual  task  knowledge  develop-ment; 
individual  TM  clarifying  data. 

3 

BO 

Make  sure  you  handle  carefully.  Cs-137 
is  an  external  gamma  hazard. 

MCkio 

Knowledge  interoperability:  TMs 
exchanging  knowledge  among  each  other. 

4 

BO 

Roger.  Will  take  precautions. 

MCkio 

Knowledge  interoperability:  TMs 
exchanging  knowledge  among  each  other. 

5 

SOCOM 

Does  CG  ship  have  proper  storage  area 
for  material  confiscated? 

MCitk 

Individual  task  knowledge  development: 
individual  TM  clarifying  data,  asking  for 
clarification. 

6 

SOCOM 

Search  team  will  report  size  of  material 
and  its  current  containment  condition; 
then  make  recommendations. 

MetCcu 

Team  integration  of  individual  TM 
knowledge  for  common  understanding; 
one  or  more  TMs  combine  individual 
pieces  of  knowledge  to  achieve 
common  understanding. 

Table  4.  MIO  Scenario  Communications  Coding: 

Knowledge  Interoperability  Development  and  Agreement  on  a  Final  Plan. 


MIO  Team  Communications 

Cognitive  Process  Coding 

Speaker 

Code 

1 

BO 

Negative  for  explosives  Station  2. 

MCkio 

Knowledge  interoperability:  TMs  exchanging 
knowledge  among  each  other. 

2 

LLNL 

Finally  received  RAD  data  from 
station  2. 

MCkio 

Knowledge  interoperability:  TMs  exchanging 
knowledge  among  each  other. 

3 

SOCOM 

Will  need  to  resolve  RAD 
containment  hazard  if  it  exists. 

MetCcu 

Team  integration  of  individual  TM  knowledge 
for  common  understanding;  one  or  more  TMs 
combine  individual  pieces  of  knowledge  to 
achieve  common  understanding. 

4 

DTRA 

If  you  have  plutonium,  you  need 
to  confiscate.  It’s  an  alpha 

MCkio 

Iterative  information  collection  and  analysis; 
collecting  and  analyzing  information  to  come  up 
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hazard,  but  still  must  be  handled 
carefully. 

with  a  solution  but  no  specific  solution  exists. 

5 

BO 

Roger. 

Misc 

Acknowledge  report. 

6 

DTRA 

By  the  way,  if  plutonium  is  in 
solid  metal  form,  your  team  can 
handle  safely  with  rubber  gloves 
and  a  dental  face  mask, 
depending  on  how  much  is  there. 

Macica 

Team  shared  understanding  development  - 
discussion  among  aU  team  members  on  a 
particular  topic  or  data  item.. 

7 

BO 

Talking  to  search  team  to  see  if 
this  is  within  their  capabilities  or 
if  we  will  need  outside  assets. 

MCkio 

Iterative  information  collection  and  analysis; 
collecting  and  analyzing  information  to  come  up 
with  a  solution  but  no  specific  solution  exists. 

8 

LLNL 

Hazard  is  probably  minimal,  can 
isolate  and  confiscate. 

Metes 

Team  agreement  on  a  common  solution  -  all  tern 
members  agree  on  the  final  plan. 

DISCUSSION 

Differences  between  the  two  scenarios  in  terms  of  how  the  team’s  behavior  maps  to  the  model  of 
team  collaboration  were  noted.  One  difference  was  that  course  of  action  selection  during  the  air 
warfare  tasks  tends  to  be  done  less  collaboratively  than  it  is  in  other  decisionmaking  domains, 
e.g.,  a  maritime  interdiction  operation  scenario,  due  to  the  inherent  time  pressure  to  make 
decisions  and  take  actions.  Decisions  tend  to  be  made  unilaterally  by  the  tactical  action  officer  or 
the  commanding  officer,  (sometimes  these  two  collaborate)  but  do  not  typically  involve 
discussion  with  the  rest  of  the  team.  Decisions  regarding  course  of  action  selection  entailed  very 
little  collaboration  for  the  air  warfare  tasks  due  to  the  speed  of  the  potential  threat  aircraft.  When 
actions  need  to  be  taken  very  quickly  in  an  attempt  to  determine  the  intent  of  an  inbound  track, 
and  a  series  of  gradually  escalating  actions  are  required,  time  is  not  available  to  discuss 
alternative  courses  of  action. 

In  general,  the  overall  task  of  responding  to  air  warfare  scenarios  consists  of  situation  assessment 
(“what’s  going  on”)  and  action  selection  (“what  to  do  about  it”).  Klein  (1989)  found  that  when 
decisionmakers  use  a  recognition-primed  decisionmaking  strategy  to  perfonn  decisionmaking 
tasks,  usually  the  situation  itself  either  determines  or  constrains  the  response  options  and  that 
experienced  decisionmakers  make  up  to  90%  of  all  decisions  without  considering  alternatives.  If 
the  situation  appears  similar  to  one  that  the  decision  maker  has  previously  experienced,  the 
pattern  will  be  recognized  and  the  course  of  action  is  usually  immediately  obvious.  The 
recognition  primed  model  of  decisionmaking  fuses  two  processes — situation  assessment  and 
mental  simulation  (Klein,  1993).  In  the  simplest  case  the  situation  is  recognized  as  familiar  or 
prototypical,  using  feature  matching,  and  the  obvious  response  is  implemented.  In  a  more 
complex  case  the  decisionmaker  performs  a  conscious  evaluation  of  the  response,  using  mental 
simulation  to  uncover  problems  prior  to  implementing  the  response.  In  the  most  complex  case 
the  evaluation  reveal  flaws  requiring  modification,  or  the  option  is  judged  inadequate  and 
rejected  in  favor  of  the  next  most  typical  reaction. 

CONCLUSIONS 

Analysis  of  data  captured  from  teams  performing  their  tasks  in  a  collaborative  environment  can 
provide  valuable  insight  into  what  constitutes  effective  collaborative  performance.  This 


18 


understanding  can  then  be  used  to  develop  technology  to  support  this  cognitive  activity,  develop 
tools  to  reduce  cognitive  workload,  and  techniques  and  processes  to  improve  information 
exchange  among  collaborating  members.  Future  plans  include  additional  analysis  for  more 
complex  scenarios  and  analysis  of  the  contribution  made  by  providing  collaborative  tools  to 
support  teams  when  performing  these  collaborative  tasks. 
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Objective 

•  Better  understand  cognitive 
processes  employed  when 
teams  collaborate  to  solve 
problems 

Approach 

•  Analyze  team  communications 
data  using  cognitive  process 
definitions 

•  Validate  and  refine  the  model 
of  team  collaboration 
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Data  Analyzed 

•  Three  Maritime  Interdiction  Operations  (MIO)  experiments 

•  Four  Air  Warfare  scenarios 

•  Firefighters  9-11 
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Team  Collaboration  Model  Validation 


GOAL:  Understand  and  improve  effectiveness  of  team  decisionmaking  in 

complex,  data-rich  situations  by  validating  model  of  team  collaboration 
Model  of  Team  Collaboration  Defines: 

Meta-cognitive  processes  that  guide  team  collaboration 

-  Information  processing  components  the  team  performs  to  achieve 
each  collaborative  stage 

-  Communication  mechanisms  used  by  the  team  to  build  the 
necessary  knowledge 

Emphasizes  cognitive  aspects  of  collaboration  process  —  includes 
major  cognitive  processes  that  underlie  this  type  of  communication: 

(1)  Individual  knowledge  building 

(2)  Knowledge  interoperability 

(3)  Team  shared  understanding  and 

(4)  Team  consensus  (Warner,  Letsky,  &  Cowen,  2004) 

Validate  that  these  processes  exist  and  how  they  contribute  to 
team  performance  through  verbal  protocol  analysis  coding  of 
team  communications. 
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Types  of  Problem  Solving  Situations 


•  Ill-Structured  Decisionmaking  Tasks 

•  Time  Pressure 

•  Dynamic  Information 

•  High  Information  Uncertainty 


•  High  Cognitive  Workload 

•  Human  System  Interface 
Complexity 


Decision  Support  (DS)  Structure 

Multiple  DMs  with  complete  participant  interaction  at  multiple 
levels  in  the  chain  of  command. 


Team  Types 


PRMSIMTIA  TER  SCUNTIAJU 


nAdm  Cebrowski  Network  centric  warfare 


Asynchronous 
Distributed 
Culturally  Diverse 
Heterogeneous  Knowledge 
Unique  Roles 
Command  Structure 
Rotating  Team  Members 


Operational  Tasks 

•  Team  Data  Processing 

•  Developing  Shared  Situational  Awareness 

•  Team  Decisionmaking  and  Course  of  Action  Selection 
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Problem  Area 


NAV^AIR 

Office  of  Naval  Research 

Collaboration  and  Knowledge  Management  (CKM)  Program 

MODEL  OF  TEAM  COLLABORATION 

Focus  on  Macro-Cognition  (September,  2005) 


SPAWAR 

V 


Systems  Center 
Sen  Diego 


3  Characteristics 


Collaborative  Situation 


Parameters: 


•  time  pressure 

•  information/knowledge 
uncertainty 

•  dynamic  information 

•  large  amount  of  knowledge 
(cognitive  overload) 

•  human-agent  interface 


Team  Tvoes 


•  asynchronous 

•  distributed 

•  culturally  diverse 

•  heterogeneous  knowledge 

•  unique  roles 

•  command  structure 
(hierarchical  vs.  flat) 

•  rotating  team  members 

Operational  Tasks 


•  team  decision  making,  COA 
selection 

•  develop  shared  understandir 

•  intelligence  analysis 
(team  data  processing) 


Collaboration  Stages  &  Cognitive  Processes 


Meta-Cognitive: 

•  individual  conversion  of 
data  to  knowledge 


Macro-Cognitive: 


•  team  integration  of  individual 
knowledge  for  common  understanding 


■  team  agreement  on  a  common 
solution 


1  solution  adjustment  to 


•  individual  mental  model 
construction 

•  knowledge  interoperability 
development 

•  individual  task,  team  and  domain 
knowledge  development 

•  individual  knowledge  object 
development 

•  individual  visualization  and 
representation  of  meaning 


■  knowledge  interoperability 
development 

■  iterative  information  collection 

and  analysis 

■  team  shared  understanding 
development 

■  develop,  rationalize,  &  visualize 
solution  alternatives 

■  convergence  of  individual  mental 
models  to  team  mental  model 

■  individual  task,  team  and  domain 
knowledge  development 


■  team  negotiation  of  solution  | 


fit  goals  and  exit  criteria 


alternatives 

■  team  pattern  recognition 

■  team  shared  understanding 
development 

■  convergence  of  individual 
mental  models 

■  critical  thinking 

■  sharing  hidden  knowledge 

■  individual  task  knowledge 
development 

■  team  task  knowledge 
development 


■  compare  problem  solution 
against  goals 

■  team  shared  understanding 
development 

■  convergence  of  individual 
mental  models  of  solution 

■  analyze,  revise  output 
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Mechanisms  for  achieving  Meta  and  MacroCognitive  Processes  (applies  to  all  stages) 

•  Verbal  communications:  representing  and  discussing  individual  information,  discussing  team  generated  information, 
questioning,  agreeing  /  disagreeing,  negotiating  perspectiyes,  discussing  possible  solutions,  providing  rationale. 

*  Non-Verbal  communications:  facial  expressions,  voice  clues  (vocal  paralanguage),  hand  gestures,  body  movements(kinJ?i 
touch  (haptics),  personal  space,  drawing,  text  messages,  augmented  video,  affordances(cognition  in  objects). 


si 


Method 


•  Verbatim  transcripts  analyzed  from  two  series  of  exp’ts  and  one  real- 
world  event  where  teams  collaborated  to  solve  a  complex  problem 

-  Maritime  Interdiction  Operations  (MIO) 

-  Air  warfare  decisionmaking 

-  Firefighters  from  9-11 

•  In  all  three  problem-solving  tasks,  assessment  is  difficult  because 
available  information  is  often  incomplete  or  ambiguous. 

-  Transcripts  include  communications  between  all  team  members 
and  decisionmakers  at  distributed  sites. 

•  Analyze  and  code  team  communications  data  using  the  cognitive 
process  definitions  developed  by  Warner,  Letsky,  &  Cowen,  2004. 

-  Focus  of  collaboration  model  is  on  knowledge  building  among 
team  members  and  developing  team  consensus  for  selection  of 
a  course  of  action 

-  Builds  on  previous  work  to  validate  model  (Warner,  et  al,  2004) 

-  Similar  methodology  applied  to  three  different  DMg  tasks 
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Experiment  I:  Maritime  Interdiction  Operations 


•  Tech’l/oper’l  challenges  of  developing  global  Maritime  Domain  Security 

-  Wireless  network  for  data  sharing  during  MIO  to  facilitate  reachback  for 
radiation  source  analysis  and  biometric  data  analysis 

-  Networking  solutions  for  MIO  where  subject  matter  experts  at 
geographically  distributed  command  centers  collaborate  with 
boarding  party  in  near  real  time  to  facilitate  SA  /  COA  selection 

•  Evaluate  networks,  adv’d  sensors,  and  collaborative  tech’y  for  rapid  MIC 

-  Rapidly  set  up  ship-to-ship  communications  that  permit  them  to 
search  for  radiation/  explosive  sources  while  maintaining  contact 
with  mothership,  C2  organizations,  and  collaborating  with  remotely 
located  sensor  experts 

•  Boarding  team  boards  suspect  vessel,  establishes  collaborative  networl 
and  begins  inspections  and  data  collection  process 

-  Boarding  officer  boards  vessel  with  his  laptop  so  he  can  collaborate 
with  all  other  members  of  the  team 

-  Co-located  on  the  ship,  physically  spread  out  (searching  for 
contraband  material  and  obtaining  fingerprints  of  crew  members) 

-  Virtual  members  of  the  boarding  team  -  experts  at  reachback  centers 

-  Commercial  uses  for  certain  radioactive  sources,  positive 

identification  of  the  source  in  a  short  time  is  imperative  8 

-  Pressure  to  conduct  the  MIO  quickly  so  as  to  not  detain  the  ship 


MIO  Team  Members 


mMsuNTArtRMfNrm, 


*  Members  of  the  boarding  team 

-  Boarding  Officer,  a  Coast  Guard  officer 

-  Representative  from  Lawrence  Livermore  National 
Labs  (LLNL)  with  portable  radiation  detection 
devices  and  “reach-back”  capability  to  LLNL 

-  Representative  from  the  Defense  Threat  Reduction 
Agency  (DTRA),  who  uses  biometrics 
measurements  of  fingerprints  and  video  imagery  to 
be  checked  against  databases  at  the  remote  facility 

-  Representative  from  Special  Operations  Command 
(SOCOM),  who  provides  guidance  on  handling 
hazardous  material. 


Maritime  Interdiction  Operations  Scenario 


V^tAlSUNTIA  HR 


•  US  Coast  Guard  ordered  cutter  to  stop,  board,  and  search 
commercial  vessel  of  foreign  origin  suspected  of  transporting 
uranium  enriching  equipment 

•  Boarding  party  brings  radiation  detection/  biometric  gear,  drawings 
of  dangerous  equipment  and  people,  and  video  recording  capability 

•  Data  collected  on  suspicious  material,  equipment,  and  people  and 
sent  to  specific  experts  at  distributed  reachback  centers 

•  Groove  collaborative  workspace  brought  expert  services  into  the 
boarding  party  team’s  tool  set 

-  Facilitated  voice  and  text  communications  between  all  members 
of  the  virtual  boarding  party  and  physical  boarding  party 

•  Requests,  transmitted  by  text  message  --  taken  for  action,  and 
radiation  source  spectrum  captures  were  made  of  suspect 
containers  that  were  detected  to  have  a  radiation  signature  presence 

•  Analysis  led  BO  to  recommend  vessel  be  quarantined  for  further 
inspection 

•  Biometric  team  took  digital  prints  of  the  crew  to  be  compared  to 
known  criminal  prints  and  latent  prints  from  terrorist  and  crime 
scenes 
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Air  Warfare  Decisionmaking 


•  Air  warfare  DMg  -  conducted  in  combat  information  center  of  Navy  ship 

•  Identification  of  large  number  of  air  tracks  under  high  time  pressure 

-  Multiple  hypotheses  regarding  threat  level  posed  to  the  battlegroup 
due  to  high  level  of  ambiguity  associated  with  the  data 

-  Nature  of  the  data,  complex  judgments  required,  and  socio- 
technical  environment  characterized  by  high  workload,  and  high 
stakes,  create  challenging  problem  for  the  air  warfare  team 

•  Incoming  info  arrives  via  various  sensor  systems  (radar,  electronic 
sup’t  measures  system,  identification  friend  or  foe,  etc.),  various 
reports,  e.g.,  intell,  other  platforms  in  area  pass  messages  regarding 
situation 

•  Reports  passed  to  rest  of  team  over  any  of  several  comm’s  systems 

-  Heard  by  all  team  members,  reports  typically  addressed  to 
specific  team  member/s,  sometimes  addressed  to  “all” 

-  Communications  passed  as  soon  as  information  is  received; 
updated  reports  are  passed  as  soon  as  new  information  is  obtained 

•  Reports  on  specific  tracks  interleaved  with  reports  on  other  tracks 

-  In  a  series  of  speech  turns,  five  separate  contacts  may  be 
discussed  at  various  levels  -  initial  reports,  updated  reports, 
sharing  information  on  response/  lack  of  response,  by  contact  fp 
action  taken  by  the  ship,  etc. 


Air  Warfare  Team  Members 


•  Six  collocated  team  members  consisted  of 

-  Commanding  officer  (CO) 

-  Tactical  action  officer  (TAO) 

-  Air  warfare  coordinator  (AAWC) 

-  Electronic  warfare  supervisor  (EWS) 

-  Identification  supervisor  (IDS) 

-  Tactical  information  coordinator  (TIC) 

•  Combat  information  center  team  members  also  communicate 
with  several  non-collocated  information  sources 

-  Battle  group  commander 

-  Saudi  air  tower 

-  Assets  passing  intelligence  reports 

-  Other  ships  and  friendly  aircraft  in  the  vicinity  of  the  battle 
group 

•  Gather  additional  information,  keep  them  apprised  of  the 
unfolding  scenario  as  they  collaborated  to  identify  air  tracks. 
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Air  Warfare  Decisionmaking 


•  Identification/  responding  to  numerous  air  contacts:  CIC  personnel 
work  as  a  team  to  identify/  determine  if  A/C  poses  a  threat 

-  High  ambiguity  often  makes  threat  assessment  a  very  difficult  task 

-  Many  pieces  of  data  fit  multiple  hypotheses 

-  Global  response  choices  (engage,  monitor,  do  nothing)  largely 
determined  by  ship’s  orders  and  the  current  geopolitical  situation 

-  Specific  actions  (e.g.,  change  course,  issue  verbal  warnings, 
illuminate  with  radar,  challenge  with  other  sensors,  etc.)  depend  on 
local  conditions,  relative  positions  of  the  inbound  contact  and  ship 

•  Determining  which  actions  is  likely  to  be  effective  depends  on 
maintaining  an  accurate  assessment  which  requires  continually 
updating  based  on  iterative  situation  assessments 

•  Critical  contacts  ident’d  based  on  ambiguous  info,  undertime  pressure 

•  High  mental  workload  --  constant  stream  of  info,  must  be  continuously 
evaluated,  e.g.,  when  info  often  pertains  to  several  different  contacts 

•  Teams  assess,  compare,  and  resolve  conflicting  info,  make  difficult 
judgments  and  remember  the  status  of  several  evolving  situations 

•  Tasks  interleaved  with  other  tasks,  such  a  making  reports  to  higher 
authority  and  requesting  assets 

•  Situation  assessment  &  action  selection 
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Coding  Process 


H  rMlfl*smmy7t*rHAi7/ 


•  Cognitive  process  coding  definitions  used  to  code  speech  turns 

•  Attempted  to  develop  criteria  for  coding  schema 

•  Codification  of  the  coding  process  is  part  of  the  overall  validation  of  the 
model,  e.g.,  goal  is  to  have  high  inter-rater  reliability  between  coders 

•  Important  to  pay  attention  to  which  track  a  team  member  was  talking  about 
when  coding  the  speech  turns 

•  First  time  discuss  a  track  --  coded  as  a  2  (individual  mental  model  (I MM) 
construction  -  where  an  individual  team  member,  using  available  info, 
develops  his/  her  mental  picture  of  the  problem  situation) 

•  After  three  speech  turns  discussing  the  same  track  (typically  involving  at  least 
four,  of  the  six  or  more  team  members)  it  was  coded  as  a  4  (team  knowledge 
development  (TKM)  -  where  all  team  members  participate  in  clarifying 
information  to  build  team  knowledge 

•  Once  five-six  team  members  had  discussed  a  track,  and  at  least  4  of  the  6  team 
members  had  been  involved  in  discussing  this  particular  track,  it  was  coded  as 
a  10  -  team  shared  understanding  development  -  which  includes  discussion 
among  all  team  members  on  a  particular  topic  or  data  item 

•  Exceptions  to  the  coding  criteria  include:  “All  stations,  [track  #  7010  is  a  comm- 
air.]”  --  he  is  telling  all  team  members  this  evaluation  of  the  track. 

-  Because  addressed  to  all  TMs  &  reported  a  higher  level/  more  final  assess’t 
of  the  track,  i.e. ,  a  comm-air,  was  coded  as  a  10.  As  more  TMs  discuss 
contact  (i.e.,  more  reports  and/or  updates  have  been  shared  among  TMs), 
cognitive  process  coding  category  reflects  a  higher  level  of  team 
understanding  of  the  situation  14 


New  Coding  Categories 


•  Issuing  an  order  regarding  a  course  of  action  --  person  with  higher  rank 

•  Tells  them  to  take  some  specific  action  against  a  potential  threat  track. 

-  Issuing  verbal  warnings,  illuminating  or  locking-on  with  radar, 
developing  a  firing  solution,  covering  with  missiles,  etc. 

-  Includes  responding/  reporting  have  taken  the  action/acknowledging 

•  Reguest  a  team  member  take  some  action  --  tell  team  member  to  do  something 

•  Not  a  direct  action  against  a  threat  track. 

-  “Can  you  try  and  change  7006  and  7005  to  assumed  hostile.” 

•  Prodding  a  team  member  to  jog  their  awareness 

•  To  make  sure  they  are  following  the  discussion 

•  Push  or  suggest  to  one  or  more  team  members  to  go  out  and 
generate  knowledge,  e.g.,  “You  should  go  back  and  see  if  there  is  ...”. 

•  Might  act  in  a  role  as  teacher  gently  pushing  collaborative  effort  certain  way 

•  “ Contrarians ”  when  a  person  says  “Let’s  re-evaluate/  reconsider 

-  Person  disagrees  with  the  current  thinking  of  the  team 

-  “Outlier”  who  makes  the  team  consider  another  viewpoint,  or 

-  “Pulls  back  the  reins” 
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Excerpt  from  MIO  Scenario  Communications  Coding: 
Developing  Solution  Alternatives 


MIO  Team  Communications 

Cognitive  Process  Coding 

Speaker 

Code 

1 

DTR 

A 

Cesium  137  can  be  used  to  make  an  RDD. 
If  there  are  no  explosives,  then  it  is  not 
configured  as  a  weapon  yet.  Recommend 
material  be  confiscated. 

sa 

itk 

Develop,  rationalize  and  visualize 
solution  alternatives'*  using  data  to 
justify  a  solution 

Individual  task  knowledse  development; 

2 

BO 

Roger  will  confiscate. 

itk 

Individual  task  knowledse  development; 
individual  TM  clarifying  data. 

3 

BO 

Make  sure  you  handle  carefully.  Cs- 
137  is  an  external  gamma  hazard. 

kio 

Knowledse  interoperability:  TMs 
exchanging  knowledge  among  each  other. 

4 

BO 

Roger.  Will  take  precautions. 

kio 

Knowledse  interoperability:  TMs 
exchanging  knowledge  among  each  other. 

5 

SOC 

OM 

Does  CG  ship  have  proper  storage 
area  for  material  confiscated? 

itk 

Individual  task  knowledse  development: 
individual  TM  clarifying  data,  asking  for 
clarification. 

6 

SOC 

OM 

Search  team  will  report  size  of 
material  and  its  current  containment 
condition;  then  make 
recommendations. 

cu 

Team  integration  of  individual  TM 
knowledge  for  common  understandins: 
one  or  more  TMs  combine  individual 
pieces  of  knowledge  to  achieve  ^  g 

common  understanding. 

p  jj£j  MIO  Scenario  Communications  Coding:  Knowledge 

Interoperability  Development  and  Agreement  on  a  Final  Plan 


MIO  Team  Communications 

Cognitive  Process  Coding 

Speaker 

Code 

BO 

Negative  for  explosives  Station 
2. 

kio 

Knowledge  interoperability :  TMs  ex- 
changing  knowledge  among  each  other. 

LLNL 

Finally  received  RAD  data 
from  station  2. 

kio 

Knowledge  interoperability :  TMs 
exchanging  knowledge  among  each 
other. 

SOCO 

M 

Will  need  to  resolve  RAD 
containment  hazard  if  it  exists. 

cu 

Team  integration  of  individual  TM 
knowledge  for  common  understanding ; 
one  or  more  TMs  combine  individual 
pieces  of  knowledge  to  achieve 
common  understanding. 

DTRA 

If  you  have  plutonium,  you 
need  to  confiscate.  It’s  an 
alpha  hazard,  but  still  must  be 
handled  carefully 

ica 

Iterative  information  collection  and 
analysis ;  collecting  and  analyzing 
information  to  come  up  with  a  solution 
but  no  specific  solution  exists . 

BO 

Roger. 

Misc 

Acknowledge  report. 
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MIO  Scenario  Communications  Coding: 

Knowledge  Interoperability  Development  and  Agreement  on 

a  Final  Plan  (cont’d) 


DTRA 

By  the  way,  if  plutonium  is 
in  solid  metal  form,  your 
team  can  handle  safely  with 
rubber  gloves  and  a  dental 
face  mask,  depending  on 
how  much  is  there. 

tsu 

Team  shared  understanding 
development  -  discussion  among  qU 
team  members  on  a  particular  topic  or 
data  item . 

BO 

Talking  to  search  team  to 
see  if  this  is  within  their 
capabilities  or  if  we  will 
need  outside  assets. 

ica 

Iterative  information  collection  and 
analysis;  collecting  and  analyzing 
information  to  come  up  with  a  solution 
but  no  specific  solution  exists. 

LLNL 

Hazard  is  probably 
minimal,  can  isolate  and 
confiscate. 

cs 

Team  agreement  on  a  common  solution 
-  all  tern  members  agree  on  the  final 
plan. 
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Air  Warfare 

MIO 

Fireflght 

Scenarios 

Scenarios 

Ulg 

Macro-Cognitive  Process  Coding 

Seen 

D- 

Seen 

D- 

CG- 

59 

DDG- 

54 

Nov 

06 

June 

06 

Sept 

06 

Firefight¬ 
ers  9-11 

Categories 

Run 

A 

Run 

B 

Knowledge  Construction 

1. 

Data  to  information  (dti) 

1 

4 

- 

37 

2 

5 

- 

2 

2. 

Individual  mental  model  (imm) 

8 

11 

18 

25 

1 

7 

8 

14 

3. 

Individual  task  knowledge  development  (itk) 

25 

30 

31 

29 

35 

7 

47 

325 

B 

Team  knowledge  development  (tk) 

11 

5 

18 

1 

3 

5 

8 

210 

5. 

Knowledge  object  development  (ko) 

- 

- 

- 

- 

- 

2 

8 

0 

6. 

Visualization  and  representation  (vrm) 

- 

- 

- 

- 

- 

- 

- 

0 

■ 

Collaborative  Team  Problem  Solving 

7. 

Common  understanding  (cu) 

- 

6 

- 

2 

6 

7 

16 

8. 

Knowledge  interoperability  (kio) 

- 

5 

- 

1 

2 

- 

10 

8 

9. 

Iterative  collection  and  analysis  (ica) 

1 

11 

- 

- 

6 

4 

14 

0 

10 

Team  shared  understanding  (tsu) 

1 

17 

28 

34 

3 

2 

3 

6 

11 

Solution  alternatives  (sa) 

- 

3 

- 

- 

6 

- 

- 

13 

12 

Convergence  of  mental  models  (cmm) 

1 

- 

- 

- 

1 

- 

- 

22 

13 

Agreement  on  Common  solution  (cs) 

- 

2 

- 

- 

- 

- 

1  19 

1 

Team  Consensus 

14 

Team  negotiation  (tn) 

15 

Team  pattern  recognition  (tpr) 

16 

Critical  thinking  (ct) 

17 

Sharing  hidden  knowledge  (shk) 

18 

Solution  adjustment  against  goal  (sag) 

■ 

Outcome  Evaluation  and  Revision 

19 

Compare  solution  options  against  goals 

(CSg) 

20 

Analyze,  revise  solutions  (aro) 

21 

Miscellaneous  (misc) 

22 

Issue  order  regarding  course  of  action 

(coa) 

23 

Request  take  action  (rta) 

I 

Totals 

Air  Warfare  Scenarios 

MIO  Scenarios 

Fireflg 

hting 

- 

- 

- 

- 

4 

- 

- 

1 

- 

- 

- 

- 

- 

- 

- 

3 

- 

- 

- 

- 

- 

- 

- 

3 

- 

2 

- 

- 

- 

- 

- 

5 

- 

- 

- 

- 

- 

- 

0 

- 

1 

- 

- 

- 

- 

- 

2 

- 

- 

- 

- 

- 

- 

- 

1 

38 

27 

57 

61 

6 

- 

- 

849 

7 

5 

17 

37 

- 

- 

2 

92 

3 

2 

18 

8 

1 

2 

11 

53 

96 

131 

187 

233 

73 

40 

118 

1626/ 

777 
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Model  of  Team  Collaboration: 

Validation 
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•  Codes  used  by  Firefighters 

-  19  out  of  the  23  cognitive  processes  in  the  model  (all 
codes  except:) 

•  knowledge  object  development  (ko)-  requires  pictures  and 
icons 

•  individual  visualization  and  representation  of  meaning  (vrm)- 
requires  visual  aids 

•  iterative  information  collection  and  analysis  (ica)-  collect  and 
analyze  information  without  mentioning  a  solution 

•  solution  adjustment  against  goal  and  exit  criteria  (sag)- 
compares  solution  option  against  goal  and  exit  criteria 

-  Did  not  pertain  to  FDNY  radio  communication  but  still 
pertain  to  other  team  collaboration  environments  and 
should  not  be  eliminated  from  the  collaboration  model 
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Model  of  Team  Collaboration: 

Validation 


Divide  2  hours,  21  minutes  of  recordings  problems  faced 

-  Larger  problem  of  Search  and  Evacuation  -  never  got 
to  final  stages  because  the  buildings  collapsed 

-  Broken  up  into  phases  to  represent  the  mental  model 
within  which  the  FDNY  was  working 

-  Divide  into  smaller  problems 

1.  What  happened?  Create  a  mental  model 
Time  period:  0846-0902 

1.  Evacuate  South  Tower  after  the  North  had  been  hit? 

Time  period:  0902-0958 

1.  How  to  divide  units  between  the  two  towers? 

Time  period:  0958-1028 

1.  Evacuate  the  North  Tower  after  the  South 
collapsed? 

Time  period:  1028-1107 
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Model  of  Team  Collaboration:  Validation 


•  Trends  in  the  codes,  1620  total  speech  turns 

-  849  (52.41%)  miscellaneous,  removed  for  the  following  percentages 
leaving  771  total  codes 

-  325  (42.15%)  itk  -  iterative  team  knowledge  development 

•  asking  lots  of  questions,  how  to  alleviate  questions  and  therefore 
message  traffic? 

-  210  (27.24%)  tk  -  Developing  team  knowledge 

•  Sharing  knowledge  with  fellow  firefighters  and  passing  knowledg 
back  to  the  dispatcher 

-  92  (11.93%)  coa  -  Course  of  action 

•  Telling  the  dispatcher  and/or  other  responding  units  what  to  do 

-  53  (6.87%)  rta  -  Request  take  action 

•  Requesting  something  of  the  dispatcher  or  responding  units 

-  22  (2.85%)  cmm  —  Constructing  team  mental  model 

-  16  2.08%)  cu  -  Developing  common  understanding 

-  14  (1 .82%)  imm  -  individual  mental  model 

•  Individuals  contributing  to  the  team’s  mental  model 


23 


self, STM, »,  jJ 


Model  of  Team  Collaboration:  Validation 


*  Inter-rater  Reliability  Analysis 

-  Two  coders,  test  subjectivity  of  model’s  codes 

-  34  out  of  1626  codes  (4.37%)  were  disagreements 

*  Discussed  differing  opinions  to  reach  an 
agreement  with  the  other  coder 

-  49  out  of  1626  codes  (6.31%)  were  decided  upon  after 
a  discussion  between  the  coders 

*  One  or  both  of  the  coders  was  unsure  of  how  to 
code  the  communication  turn  and  left  it  to  discuss 
further  with  the  other  coder. 

*  In  total,  did  not  completely  agree  on  10.68%  of  codes 

*  Reliable  89.32%  of  the  codes 
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Adherence  to  SOP 


*  Minor  Deviations,  SOP  Deviation  #1 

-  ID  speaker  and  addressee 

-  Requesting  ambulances  and  units 

-  Casual  communication 

-  10  codes 

*  Major  Deviations,  SOP  Deviation  #2 

-  Unit  to  Unit  transmissions 

-  Use  first  names 

*  Major  Deviations,  SOP  Deviation  #3 

-  Urgent  Radio  Messages  (24  messages  identified 
as  urgent) 

-  Mayday  Radio  Messages  (3  messages  identified 
as  mayday) 

*  Department-wide  Recall 

-  Never  used  before 

-  Unclear  as  to  where  to  go,  what  to  do 
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Discussion 

Differences  between  three  scenarios  -  how  the  team’s  behavior  maps  to  the  model 

-  Course  of  action  selection  is  done  less  collaboratively  in  tactical  domains, 
due  to  inherent  time  pressure  to  make  decisions  and  take  actions 

•  Decisions  made  unilaterally  by  tactical  action  officer  or  commanding  officer  -- 
do  not  typically  involve  discussion  with  the  rest  of  the  team. 

•  Decisions  regarding  course  of  action  selection  entailed  very  little  collaboration  for 
air  warfare  tasks  due  to  the  speed  of  the  potential  threat  aircraft. 

-  When  actions  need  to  be  taken  very  quickly  in  an  attempt  to  determine  the  intent 
of  an  inbound  track,  time  is  not  available  to  discuss  alternative  courses  of  action 

Air  warfare  consists  of  situation  assessment  (“what’s  going  on”)  and  action  selection 
(“what  to  do  about  it”) 

-  Decisionmakers  use  a  recognition-primed  decisionmaking  strategy  (Klein,  1989) 

•  Situation  itself  either  determines  or  constrains  the  response  options 

•  Recognition  primed  model  of  decisionmaking  fuses  two  processes  — 
situation  assessment  and  mental  simulation  (Klein,  1993) 

-  Simplest  case  the  situation  is  recognized  as  familiar  or  prototypical, 
using  feature  matching,  and  the  obvious  response  is  implemented 

-  More  complex  case  --  decisionmaker  performs  conscious  evaluation  of 
response,  using  mental  simulation  to  uncover  problems  prior  to 
implementing 

-  In  most  complex  case  --  evaluation  reveals  flaws  requiring  modification, 
or  option  is  judged  inadequate/rejected  in  favor  of  next  typical  reaction 

Experienced  DMs  make  90%  of  all  decisions  w/o  considering  alternatives 

•  If  situation  appears  similar  to  one  previously  experienced,  pattern  will  b$j 
recognized  and  COA  is  usually  immediately  obvious 


Adherence  to  SOP-  Mayday 
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Type  of  Mayday  Message 

FDNY  Communications 

Speaker 

Imminent  collapse  feared 

N/A 

Structural  collapse  has  occurred 

FIELD 

Engine  3-9  acting,  report  on  the  22nd  floor, 
reporting  a  floor  collapse  at  that  location,  K. 

A  firefighter  is  unconscious  or 
suffers  a  life  threatening  injury 

FIELD 

We  have  a  medical  emergency,  possible 
heart  attack,  firemen,  we're  on  the  bulkhead, 
west,  requesting  oxygen  for  the  firemen,  K. 

A  firefighter  becomes  aware 
that  another  firefighter  is  missing 

N/A 

A  firefighter  becomes  trapped  or  lost 

A  civilian  came  on  the  radio  asking  for  help 
because  they  were  trapped  in  the  rubble  after 
the  South  Tower  collapsed.  While  the 
civilian  did  not  know  about  the  correct  use  of 
“mayday”  the  dispatcher  relayed  the  message 
saying,  “transmitting  a  mayday.” 
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Adherence  to  SOP-  Urgent 


Type  of  Urgent 
Message 

FDNY  Communications 

Speaker 

A  firefighter  suffers 
an  injury  that  is  not 
life  threatening,  but 
requires  medical 
attention  and 
hospital  care 

DISPATCH 

Manhattan  to  Field  Comm.,  urgent. 

FIELD 

Receive,  Manhattan,  Field  Comm. 

DISPATCH 

Tower  No.  2,  19th  floor,  firefighter  down.  Tower  No.  2,  19th  floor,  firefighter  down. 

FIELD 

Field  Comm.  Received. 

Discovery  of  a 
structural  problem 
indicating  the  danger 
of  collapse 

FIELD 

Engine  3-9  acting,  report  on  the  22nd  floor,  reporting  a  floor  collapse  at  that  location,  K. 

FIELD 

Marine  1  to  Manhattan  with  an  urgent  message,  K. 

DISPATCH 

Unit  with  an  urgent  message,  K. 

Fire  is  entering  an 
exposure  to  a  degree 
that  any  delay  may 
considerably  enlarge 
the  fire  problem 

FIELD 

This  is  Marine  1,  we’re  in  the  river.  You’ve  got  fire  out  of  the  north  side  and  now 
coming  out  of  the  west  side  of  the  World  Trade  Center,  the  west  side. 

Report  of  apparatus 
breakdown  while 
unit  is  responding  to 
an  alarm 

FIELD 

Engine  3 1 7  to  Manhattan,  urgent. 

DISPATCH 

Engine  3-1-7,  go. 

FIELD 

I’ve  got...  from  the  Port  Authority  telling  me  that  the  elevators  are  on  the  44th  floor. 
Don’t  use  them,  they’re  about  to  come  down. 

Loss  of  water  which 
would  endanger 
firefighters 

DISPATCH 

Engine  33  urgent,  go. 

FIELD 

Engine  22  is  being  manned  by  an  off-duty  member  form  Rescue  1 .  Be  advised  it  appears 
that  we  have  lost  water  pressure  down  in  lower  Manhattan.  Can  you  have  Ma^e  1  or 
any  other  available  fire  boat  respond  to  Vescey  Street  on  the  West  Side?  We're  going  to 
need  water  supply  into  the  area,  K. 

Loss  of  SA 


•  Minor  Losses 

-  Vague,  inaccurate  information 

*  Reporting  floor  numbers 

*  Referring  to  the  two  towers 

*  Major  Losses 

-  After  the  South  Tower  collapsed 

*  Who  survived?  Field  Comm? 

-  After  the  North  Tower  collapsed 

*  Where  were  the  responding  units?  Who  was  in 
which  tower? 

Made  rescuing  those  trapped  very  difficult 
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Loss  of  SA 
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SITUATIONAL  AWARENESS  LOST 

FIX 

EFFECTS 

TIME 

SPEAKER 

MESSAGE 

0904 

Marine  6 

Marine  6,  that  plane  was  a  large  bomber-style  green 
aircraft  into  the  second  tower,  be  advised. 

None. 

None. 

0913 

Car  9 

Car  9  to  Manhattan. 

Car  9  came 
back  on  the 
radio  to  correct 
Dispatch,  saying 
they  wanted  the 
mobile  command 
vehicle,  not  field 

com 

None,  corrected  before  it 
had  an  impact. 

DISPATCH 

Car  9  go  ahead. 

Car  9 

Would  you  advise  the  mobile  command  vehicle  to  come  in 
on  West  and  Liberty  Street,  West  and  Liberty  Street. 

DISPATCH 

I  already  advised  them. 

Car  9 

What’s  their  ETA? 

DIS¬ 

PATCH 

Manhattan  calling  Field  Comm. 

0930 

Engine 

317 

I've  got ...  from  the  Port  Authority  telling  me  that  the 
elevators  are  on  the  44th  floor.  Don't  use  them, 
they're  about  to  come  down. 

None. 

The  firefighters  work¬ 
ing  in  whichever  build¬ 
ing  the  elevators  were 
not  coming  down  in 
would  have  had  con¬ 
tinued  access  to 
elevators,  but  instead 
were  told  not  to  use 
them  because  of  ^gue 
information. 

DIS¬ 

PATCH 

Is  that  going  to  be  for  No.  2  or  No.  1 
World  Trade? 

Wasn’t  sure.  I’d  say  go  with  both. 

DIS¬ 

PATCH 

Attention  all  companies  operating  at  the  fifth  alarm  for  both 
World  Trade  Centers,  the  elevators,  the  Port  Authority 
reports  the  elevators  on  the  No.  4-4  floor  are  about  to  come 
down.  All  companies  operating  at  No.  1  and  No.  2  World 
Trade  Center  at  the  fifth  alarm,  do  not  use  the  elevators. 

They  are  about  to  come  down  as  per  the  Port  Authority  on 
the  No.  4-4  floor.  Field  Comm.,  receive  that  urgent? 
Manhattan  to  Fadder  2- 1 ,  K. 
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